Skip to content

Trainer with Iterable Dataset #7858

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
sgugger merged 7 commits into from
Oct 19, 2020
Merged

Trainer with Iterable Dataset #7858

sgugger merged 7 commits into from
Oct 19, 2020

Conversation

@j-rossi-nl
Copy link
Contributor

@j-rossi-nl j-rossi-nl commented Oct 16, 2020

What does this PR do?

Fixes #5990
Follows #5995 (closed because stale).
Merges all commits from master.

  • accomodate iterable dataset without predefined length
  • set it as 1 use case: provide max_steps, and NO num_epochs
  • Is a merge of master and PR 5995
  • This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to the it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

@sgugger

@j-rossi-nl
fix 5990
dea9722 
* accomodate iterable dataset without predefined length
* set it as 1 use case: provide max_steps, and NO num_epochs
* Is a merge of master and PR 5995
@j-rossi-nl
fix trainer test under TF
fa1fe64 
* fix only for torch
* TF trainer untouched
* trainer tests are skipped when no torch
@j-rossi-nl

This comment has been minimized.

Sign in to view
@j-rossi-nl

This comment has been minimized.

Sign in to view
sgugger
sgugger suggested changes Oct 17, 2020
View reviewed changes
Copy link
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your PR. There are a few things I'd like to adjust, on the error messages especially. Or the comments in the tests.

In general, please limit your PR to the functionality you are adding. We can discuss unrelated changes to other parts of the code in separate PRs but it makes reviewing less efficient if you try to group them together.

@j-rossi-nl j-rossi-nl requested a review from sgugger October 18, 2020 10:22
sgugger
sgugger suggested changes Oct 19, 2020
View reviewed changes
Copy link
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing most of the items in the review. I still disagree with your changes in the test file, which makes reusing the RegressionDataset for other frameworks impossible (and is unrelated to the feature you are trying to implement).

@j-rossi-nl
remove torch.dataset from test_trainer
7f27999 
* unnecessary inheritance
* RegressionDataset implements all needed methods __len__ and __getitem__
@j-rossi-nl
fix quality checks
649bf8e
@j-rossi-nl j-rossi-nl requested a review from sgugger October 19, 2020 15:04
sgugger
sgugger reviewed Oct 19, 2020
View reviewed changes
@j-rossi-nl
restore RegressionDataset
1fde7d6 
* was wrongly under is_torch_available()
LysandreJik
LysandreJik approved these changes Oct 19, 2020
View reviewed changes
Copy link
Member

@LysandreJik LysandreJik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sgugger sgugger merged commit a09fe14 into huggingface:master Oct 19, 2020
@sgugger
Copy link
Collaborator

sgugger commented Oct 19, 2020

Thanks for your contribution!

@j-rossi-nl j-rossi-nl mentioned this pull request Oct 19, 2020
Closed
This was referenced Oct 27, 2020
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@LysandreJik LysandreJik LysandreJik approved these changes

+1 more reviewer

@sgugger sgugger sgugger requested changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Trainer: exception raised when calling len() on IterableDataset

3 participants

@j-rossi-nl @sgugger @LysandreJik